The Importance of CICD and the Comparison between Jenkins and Github Action

70% of Failures Occur During Changes

Today, when reading “SRE: Google’s Workbook for Reliable Systems”, a sentence struck a chord with me. It roughly goes like this: About 70% of failures in a service occur during changes.

I don’t know how they came up with the figure of 70%. However, from my personal experience as both a developer and an operator, the point behind this statement is correct: Changes are one of the main sources of failures.

When I first started working, the company I was in had no CICD system at all. Operations staff had to manually type commands to release changes, and the failure rate during that period was the highest in my working career.
Especially when a new system was released, it was common to face the situation where after an overnight release process, it had to be rolled back at 6 or 7 am in the morning.
Another situation was when making changes to a large system with fifty or sixty servers. The same steps had to be executed dozens of times on different servers. This not only wasted time, but more importantly, repeating the same thing dozens of times was prone to errors, ultimately leading to failed or delayed changes.

When I was about to leave the company, other colleagues set up a Jenkins for CICD and wrote pipelines in advance. This had two benefits:

1. It didn’t rely on operations colleagues to manually execute commands, which both reduced errors and saved execution time.
2. Since the pipelines were written in advance, to a certain extent, it restricted and standardized what developers should do during the release process. In this way, developers wouldn’t ask operations to manually complete tasks that should have been done through code or configuration files, as they did during the manual release stage. At that time, I clearly felt that the change failure rate had decreased significantly, and the service availability naturally increased.

How to Choose between Jenkins and Github Action?

I started getting in touch with Jenkins the year before last. From deployment to maintenance and actual use, I can be considered quite familiar with it. Until I used Github Action this year, I always thought that Jenkins was really a great piece of software (I still think so, but I’ve discovered some of its drawbacks).

Here are three articles about the comparison between Jenkins and Github Action that I think are quite good:

1. The official comparison by Github
   https://docs.github.com/cn/actions/learn-github-actions/migrating-from-jenkins-to-github-actions

2. A personal blog on codecentric
   https://blog.codecentric.de/en/2021/03/github-actions-nextgen-cicd/

3. A blog on CSDN
   https://blog.csdn.net/M2l0ZgSsVc7r69eFdTj/article/details/112975233

From the year before last until now, I have successively participated in projects to build CICD systems based on Jenkins and Github Action. Besides the comparisons from aspects such as technology and cost in the above blogs, I’ve also observed some things on my own, mainly in the following aspects:

1. Do You Currently Have a CICD System?

If the project or company already has a relatively complete CICD system that can meet most of the requirements, and there are no major changes foreseeable in the future, then sticking to the current system is probably the best choice.

After all, whether it’s CICD selection or system architecture selection, they are all in the service of “business”. The business is the truly important or valuable part. We shouldn’t put the cart before the horse and make changes simply for the pursuit of technology.

2. Is Your SCM System Github Enterprise Edition?

After Microsoft acquired Github, it has done a lot. There is a clear trend that Github is moving towards “all in one”.

Github Enterprise Edition is no longer just a simple SCM system. It has started to dabble in areas such as CICD and various security scans, and has begun to deeply integrate with Azure. Since Github is originally a place to store code, when it starts to engage in these areas, it naturally has many advantages over third - party applications. For example, there is no need to take the token outside of Github, and just this one point greatly reduces the risk of code leakage.

If you are using Github Enterprise Edition, then most likely you are a large company and should also be using AWS or Azure at the same time. In this case, it may be a better choice to start using Github Action as soon as possible. Because now Github Action is starting to do some deep integration work with foreign public clouds. Considering the “all in one” trend, Github may very likely become a one - stop solution for the code ecosystem in the future, with everything available to meet most of the requirements, making it very convenient to use.

3. Is Your CICD Process Very Complex?

Although Github Action is very convenient, it still lacks some functions compared to Jenkins. For example, result visualization, special compilation environments (such as Solaris), etc.
Take a typical web project as an example. Common CICD may include several major steps such as security scanning, code compilation, unit testing, and release to the corresponding environment. Such a CICD process can probably be easily handled by either Jenkins or Github Action.
However, if a project’s CICD process has dozens of pipelines or workflows, and they are interdependent, and even uses systems like Solaris or needs to integrate the CICD results into your own project, then currently, Jenkins is more suitable. As an old - fashioned CICD system, Jenkins has a very rich set of plugins, and since it is open - source, it is very convenient to do some deep integration.

4. Does Your Company Have High Security Requirements?

As we all know, Jenkins itself and its plugins are often exposed with various vulnerabilities, even very serious ones. In fact, this can’t really be blamed on Jenkins. After all, there are so many plugins and it is open - source, so it’s hard to ensure that there are no vulnerabilities. But if your company attaches great importance to security and can’t tolerate vulnerabilities, then Jenkins may not be a good choice for you. Because using Jenkins may very likely mean constantly having headaches about how to fix vulnerabilities, and sometimes the cost of fixing vulnerabilities can be very high. For example, due to a plugin, all the pipelines of projects using this plugin have to be rewritten, and at this time, the complaints from the R & D team may overwhelm you.
However, having vulnerabilities doesn’t mean that Jenkins can’t be used. If the company’s security requirements, especially for internal systems, are not that high, in many cases, the threats brought by vulnerabilities can be reduced or even avoided through network restrictions and permission restrictions. It may be easier to control when the team size is not very large.

Many companies in China probably use GitLab instead of Github, after all, Github is really expensive. In fact, the comparison between GitLab CI and Jenkins can also be roughly referred to the above aspects.